An ensemble learning approach for the classification of remote sensing scenes based on covariance pooling of CNN features

International audienceThis paper aims at presenting a novel ensemble learning approach based on the concept of covariance pooling of CNN features issued from a pretrained model. Starting from a supervised classification algorithm, named multilayer stacked covariance pooling (MSCP), which exploits simultaneously second order statistics and deep learning features, we propose an alternative strategy which employs an ensemble learning approach among the stacked convolutional feature maps. The aggregation of multiple learning algorithm decisions, produced by different stacked subsets, permits to obtain a better predictive classification performance. An application for the classification of large scale remote sensing images is next proposed. The experimental results, conducted on two challenging datasets, namely UC Merced and AID datasets, improve the classification accuracy while maintaining a low computation time. This confirms, besides the interest of exploiting second order statistics, the benefit of adopting an ensemble learning approach

Encodage de matrices de covariance par les vecteurs de Fisher log-euclidien : application à la classification supervisée d'images satellitaires

National audienceThis paper introduces a new hybrid architecture based on Fisher vector encoding (VF) of the convolutional layer outputs of a neural network. The originality of this work is based on the exploitation of second-order statistics via the calculation of local covariance matrices. Considering the intrinsic properties of the Riemannian manifold of covariance matrices, we propose to use the log-euclidean metric in order to extend the concept of VF encoding: the log-euclidean Fisher vectors (LE VF). The proposed architecture is then evaluated on different remote sensing databases : the UC Merced Land Use Land Cover database, the AID database, as well as on two Pléiades datasets on maritime pine forests and oyster beds.Cet article présente une nouvelle architecture hybride basée sur l'encodage par vecteurs de Fisher (VF) des sorties des couches convolutives d'un réseau de neurones. L'originalité de ce travail repose sur l'exploitation des statistiques d'ordre deux via le calcul des matrices de covariance locales. Considérant les propriétés intrinsèques à la géométrie Riemannienne propre à l'espace des matrices de covariance, nous proposons d'utiliser la métrique log-euclidienne afin d'étendre le concept des VF pour l'encodage de matrices de covariance : les vecteurs de Fisher log-euclidiens (LE VF). L'architecture proposée est ensuite évaluée sur différentes bases de données de télédétection : la base UC Merced Land Use Land Cover, la base AID, ainsi que sur deux jeux de données Pléiades sur des forêts de pins maritimes et de parcs ostréicoles

Ensemble Learning Approaches Based on Covariance Pooling of CNN Features for High Resolution Remote Sensing Scene Classification

Remote sensing image scene classification, which consists of labeling remote sensing images with a set of categories based on their content, has received remarkable attention for many applications such as land use mapping. Standard approaches are based on the multi-layer representation of first-order convolutional neural network (CNN) features. However, second-order CNNs have recently been shown to outperform traditional first-order CNNs for many computer vision tasks. Hence, the aim of this paper is to show the use of second-order statistics of CNN features for remote sensing scene classification. This takes the form of covariance matrices computed locally or globally on the output of a CNN. However, these datapoints do not lie in an Euclidean space but a Riemannian manifold. To manipulate them, Euclidean tools are not adapted. Other metrics should be considered such as the log-Euclidean one. This consists of projecting the set of covariance matrices on a tangent space defined at a reference point. In this tangent plane, which is a vector space, conventional machine learning algorithms can be considered, such as the Fisher vector encoding or SVM classifier. Based on this log-Euclidean framework, we propose a novel transfer learning approach composed of two hybrid architectures based on covariance pooling of CNN features, the first is local and the second is global. They rely on the extraction of features from models pre-trained on the ImageNet dataset processed with some machine learning algorithms. The first hybrid architecture consists of an ensemble learning approach with the log-Euclidean Fisher vector encoding of region covariance matrices computed locally on the first layers of a CNN. The second one concerns an ensemble learning approach based on the covariance pooling of CNN features extracted globally from the deepest layers. These two ensemble learning approaches are then combined together based on the strategy of the most diverse ensembles. For validation and comparison purposes, the proposed approach is tested on various challenging remote sensing datasets. Experimental results exhibit a significant gain of approximately 2% in overall accuracy for the proposed approach compared to a similar state-of-the-art method based on covariance pooling of CNN features (on the UC Merced dataset)

Méthodes d'ensemble sur l'espace des matrices de covariance : application à la classification de scènes de télédétection et de séries temporelles multivariées

In view of the growing success of second-order statistics in classification problems, the work of this thesis has been oriented towards the development of learning methods in manifolds. Indeed, covariance matrices are symmetric positive definite matrices that live in a non-Euclidean space. It is therefore necessary to adapt the classical tools of Euclidean geometry to handle this type of data. To do that, we have proposed to exploit the log-Euclidean metric. This latter allows to project the set of covariance matrices on a tangent plane to the manifold defined at a reference point, classically chosen equal to the identity matrix, followed by a vectorization step to obtain the log-Euclidean representation. On this tangent plane, it is possible to define parametric Gaussian models as well as Gaussian mixture models. Nevertheless, this projection on a single tangent plane can induce distortions. In order to overcome this limitation, we have proposed a GMM model composed of several tangent planes, where the reference points are defined by the centers of each cluster.In view of the success of neural networks, in particular convolutional neural networks (CNNs), we have proposed two hybrid transfer learning approaches based on the covariance matrix computed locally and globally on the CNN convolutional layers’ outputs. The local approach relies on the covariance matrices extracted locally on the first layers of a CNN, which are then encoded by the Fisher vectors computed on their log-Euclidean representation, while for the global approach, a single covariance matrix is computed on the feature maps of the CNN deep layers. Moreover, in order to give more importance to the objects of interest present in the images, we proposed to use a covariance matrix weighted by the saliency information. Furthermore, in order to take advantage of both local and global aspects, these two approaches are subsequently combined in an ensemble strategy.On the other hand, the availability of multivariate time series has aroused the interest of the remote sensing community and more generally of machine learning researchers for the development of new learning strategies dedicated to supervised classification. In particular, methods based on the calculation of point-to-point distance between series. Moreover, two series belonging to the same class can evolve in different ways, which can induce temporal distortions (translation, compression, dilation, etc.). To avoid this, warping methods allow to align the time series. In order to extend this approach to time series of covariance matrices, while ensuring invariance to the re-parametrization of the series, we were interested in the TSRVF representation. In the same context, several ensemble methods have been proposed in the literature, including TCK, which relies on similarity computation to classify time series. We have proposed to extend this strategy to covariance matrices by introducing the SO-TCK approach which relies on the log-Euclidean representation of such matrices.Finally, the last axis of this thesis concerns the modeling of temporal trajectories of signals measured by the radar (Sentinel 1) and optical (Sentinel 2) sensors. In particular, we are interested in the forestry problem of the chestnut ink disease in the Montmorency forest. For this purpose, we developed classification and regression models to predict a health status score from the covariance matrix computed on multi-temporal radiometric attributes.Devant le succès grandissant des statistiques du second ordre dans les problèmes de classification, les travaux de cette thèse se sont orientés vers le développement de méthodes d’apprentissage sur variétés. En effet, les matrices de covariance sont des matrices symétriques définies positives qui vivent dans un espace non Euclidien. Il est donc nécessaire de réadapter les outils classiques de la géométrie Euclidienne pour manipuler ce type de données. Pour ce faire, nous avons proposé d’exploiter la métrique log-Euclidienne. Celle-ci permet de projeter l’ensemble des matrices de covariance sur un plan tangent à la variété défini à un point de référence, classiquement choisi égal à la matrice identité, suivi d’une étape de vectorisation pour obtenir la représentation log-Euclidienne. Sur ce plan tangent, il est possible de définir des modèles paramétriques Gaussien ainsi que des modèles de mélange de Gaussiennes. Néanmoins, cette projection sur un unique plan tangent peut induire des distorsions. Afin de limiter cela, nous avons proposé un modèle de GMM composé de plusieurs plans tangents, où les points de référence sont définis par les centres de chaque cluster.Au vu de la réussite remportée par les réseaux de neurones, en particulier les réseaux de neurones convolutifs (CNN), nous avons proposé deux approches hybrides d’apprentissage par transfert basées sur la matrice de covariance calculée de façon locale et globale sur les sorties des couches convolutives d’un CNN. D’une part, l’approche locale s’appuie sur les matrices de covariance extraites localement sur les premières couches d’un CNN, qui sont ensuite encodées par les vecteurs de Fisher calculés sur leur représentation log-Euclidienne. Tandis que pour l’approche globale, une seule matrice de covariance est calculée sur les cartes de caractéristiques des couches profondes d’un CNN. De plus, afin de donner une plus grande importance aux objets d’intérêt présents dans les images, nous avons proposé d’utiliser une matrice de covariance pondérée par l’information de saillance. Par ailleurs, afin de tirer profit des aspects local et global, ces deux approches sont par la suite combinées dans une stratégie d’ensemble.D'autre part, la disponibilité des séries temporelles multivariées a suscité l’intérêt de la communauté de la télédétection et plus généralement du machine learning pour l’élaboration de nouvelles stratégies d'apprentissage pour la classification supervisée, notamment les méthodes basées sur le calcul de distance point à point entre les séries. Par ailleurs, deux séries appartenant à la même classe peuvent évoluer de façons différentes, ce qui peut induire des distorsions temporelles (translation, compression, dilatation, etc.). Pour s’affranchir de cela, le « warping » permet d’aligner les séries temporelles. Afin d’étendre cette approche pour des séries temporelles de matrices de covariance, tout en assurant l’invariance à la reparamétrisation des séries, nous nous sommes intéressés à la représentation TSRVF. Dans le même contexte, plusieurs méthodes d’ensemble ont été proposées dans la littérature, notamment le TCK, qui repose sur le calcul de similarité afin de classifier les séries temporelles. Nous avons proposé d’étendre cette stratégie aux matrices de covariance en introduisant l’approche SO-TCK qui s’appuie sur la représentation log-Euclidienne de ces matrices.Finalement, le dernier axe de cette thèse concerne la modélisation de trajectoires temporelles des signaux mesurés par les capteurs radar (Sentinel 1) et optique (Sentinel 2). En particulier, nous nous sommes intéressés au problème sylvosanitaire de la maladie de l’encre du châtaignier sur la forêt de Montmorency. Pour cela, nous avons développé des modèles de classification et de régression afin de prédire une note d’état sanitaire à partir de la matrice de covariance calculée sur les attributs radiométriques multitemporels

Ensemble learning methods on the space of covariance matrices : application to remote sensing scene and multivariate time series classification

Devant le succès grandissant des statistiques du second ordre dans les problèmes de classification, les travaux de cette thèse se sont orientés vers le développement de méthodes d’apprentissage sur variétés. En effet, les matrices de covariance sont des matrices symétriques définies positives qui vivent dans un espace non Euclidien. Il est donc nécessaire de réadapter les outils classiques de la géométrie Euclidienne pour manipuler ce type de données. Pour ce faire, nous avons proposé d’exploiter la métrique log-Euclidienne. Celle-ci permet de projeter l’ensemble des matrices de covariance sur un plan tangent à la variété défini à un point de référence, classiquement choisi égal à la matrice identité, suivi d’une étape de vectorisation pour obtenir la représentation log-Euclidienne. Sur ce plan tangent, il est possible de définir des modèles paramétriques Gaussien ainsi que des modèles de mélange de Gaussiennes. Néanmoins, cette projection sur un unique plan tangent peut induire des distorsions. Afin de limiter cela, nous avons proposé un modèle de GMM composé de plusieurs plans tangents, où les points de référence sont définis par les centres de chaque cluster.Au vu de la réussite remportée par les réseaux de neurones, en particulier les réseaux de neurones convolutifs (CNN), nous avons proposé deux approches hybrides d’apprentissage par transfert basées sur la matrice de covariance calculée de façon locale et globale sur les sorties des couches convolutives d’un CNN. D’une part, l’approche locale s’appuie sur les matrices de covariance extraites localement sur les premières couches d’un CNN, qui sont ensuite encodées par les vecteurs de Fisher calculés sur leur représentation log-Euclidienne. Tandis que pour l’approche globale, une seule matrice de covariance est calculée sur les cartes de caractéristiques des couches profondes d’un CNN. De plus, afin de donner une plus grande importance aux objets d’intérêt présents dans les images, nous avons proposé d’utiliser une matrice de covariance pondérée par l’information de saillance. Par ailleurs, afin de tirer profit des aspects local et global, ces deux approches sont par la suite combinées dans une stratégie d’ensemble.D'autre part, la disponibilité des séries temporelles multivariées a suscité l’intérêt de la communauté de la télédétection et plus généralement du machine learning pour l’élaboration de nouvelles stratégies d'apprentissage pour la classification supervisée, notamment les méthodes basées sur le calcul de distance point à point entre les séries. Par ailleurs, deux séries appartenant à la même classe peuvent évoluer de façons différentes, ce qui peut induire des distorsions temporelles (translation, compression, dilatation, etc.). Pour s’affranchir de cela, le « warping » permet d’aligner les séries temporelles. Afin d’étendre cette approche pour des séries temporelles de matrices de covariance, tout en assurant l’invariance à la reparamétrisation des séries, nous nous sommes intéressés à la représentation TSRVF. Dans le même contexte, plusieurs méthodes d’ensemble ont été proposées dans la littérature, notamment le TCK, qui repose sur le calcul de similarité afin de classifier les séries temporelles. Nous avons proposé d’étendre cette stratégie aux matrices de covariance en introduisant l’approche SO-TCK qui s’appuie sur la représentation log-Euclidienne de ces matrices.Finalement, le dernier axe de cette thèse concerne la modélisation de trajectoires temporelles des signaux mesurés par les capteurs radar (Sentinel 1) et optique (Sentinel 2). En particulier, nous nous sommes intéressés au problème sylvosanitaire de la maladie de l’encre du châtaignier sur la forêt de Montmorency. Pour cela, nous avons développé des modèles de classification et de régression afin de prédire une note d’état sanitaire à partir de la matrice de covariance calculée sur les attributs radiométriques multitemporels.In view of the growing success of second-order statistics in classification problems, the work of this thesis has been oriented towards the development of learning methods in manifolds. Indeed, covariance matrices are symmetric positive definite matrices that live in a non-Euclidean space. It is therefore necessary to adapt the classical tools of Euclidean geometry to handle this type of data. To do that, we have proposed to exploit the log-Euclidean metric. This latter allows to project the set of covariance matrices on a tangent plane to the manifold defined at a reference point, classically chosen equal to the identity matrix, followed by a vectorization step to obtain the log-Euclidean representation. On this tangent plane, it is possible to define parametric Gaussian models as well as Gaussian mixture models. Nevertheless, this projection on a single tangent plane can induce distortions. In order to overcome this limitation, we have proposed a GMM model composed of several tangent planes, where the reference points are defined by the centers of each cluster.In view of the success of neural networks, in particular convolutional neural networks (CNNs), we have proposed two hybrid transfer learning approaches based on the covariance matrix computed locally and globally on the CNN convolutional layers’ outputs. The local approach relies on the covariance matrices extracted locally on the first layers of a CNN, which are then encoded by the Fisher vectors computed on their log-Euclidean representation, while for the global approach, a single covariance matrix is computed on the feature maps of the CNN deep layers. Moreover, in order to give more importance to the objects of interest present in the images, we proposed to use a covariance matrix weighted by the saliency information. Furthermore, in order to take advantage of both local and global aspects, these two approaches are subsequently combined in an ensemble strategy.On the other hand, the availability of multivariate time series has aroused the interest of the remote sensing community and more generally of machine learning researchers for the development of new learning strategies dedicated to supervised classification. In particular, methods based on the calculation of point-to-point distance between series. Moreover, two series belonging to the same class can evolve in different ways, which can induce temporal distortions (translation, compression, dilation, etc.). To avoid this, warping methods allow to align the time series. In order to extend this approach to time series of covariance matrices, while ensuring invariance to the re-parametrization of the series, we were interested in the TSRVF representation. In the same context, several ensemble methods have been proposed in the literature, including TCK, which relies on similarity computation to classify time series. We have proposed to extend this strategy to covariance matrices by introducing the SO-TCK approach which relies on the log-Euclidean representation of such matrices.Finally, the last axis of this thesis concerns the modeling of temporal trajectories of signals measured by the radar (Sentinel 1) and optical (Sentinel 2) sensors. In particular, we are interested in the forestry problem of the chestnut ink disease in the Montmorency forest. For this purpose, we developed classification and regression models to predict a health status score from the covariance matrix computed on multi-temporal radiometric attributes

Remote Sensing Scene Classification Based on Covariance Pooling of Multi-layer CNN Features Guided by Saliency Maps

The new generation of remote sensing imaging sensors enables high spatial, spectral and temporal resolution images with high revisit frequencies. These sensors allow the acquisition of multi-spectral and multi-temporal images. The availability of these data has raised the interest of the remote sensing community to develop novel machine learning strategies for supervised classification. This paper aims at introducing a novel supervised classification algorithm based on covariance pooling of multi-layer convolutional neural network (CNN) features. The basic idea consists in an ensemble learning approach based on covariance matrices estimation from CNN features. Then, after being projected on the log-Euclidean space, an SVM classifier is used to make a decision. In order to give more strength to relatively small objects of interest in the scene, we propose to incorporate the visual saliency map in the process. For that, inspired by the theory of robust statistics, a weighted covariance matrix estimator is considered. Larger weights are given to more salient regions. Finally, some experiments on remote sensing classification are conducted on the UC Merced land use dataset. The obtained results confirm the potential of the proposed approach in terms of classification scene accuracy. It demonstrates, besides the interest of exploiting second order statistics and adopting an ensemble learning approach, the benefit of incorporating visual saliency maps

Remote Sensing Scene Classification Based on Covariance Pooling of Multi-layer CNN Features Guided by Saliency Maps

The new generation of remote sensing imaging sensors enables high spatial, spectral and temporal resolution images with high revisit frequencies. These sensors allow the acquisition of multi-spectral and multi-temporal images. The availability of these data has raised the interest of the remote sensing community to develop novel machine learning strategies for supervised classification. This paper aims at introducing a novel supervised classification algorithm based on covariance pooling of multi-layer convolutional neural network (CNN) features. The basic idea consists in an ensemble learning approach based on covariance matrices estimation from CNN features. Then, after being projected on the log-Euclidean space, an SVM classifier is used to make a decision. In order to give more strength to relatively small objects of interest in the scene, we propose to incorporate the visual saliency map in the process. For that, inspired by the theory of robust statistics, a weighted covariance matrix estimator is considered. Larger weights are given to more salient regions. Finally, some experiments on remote sensing classification are conducted on the UC Merced land use dataset. The obtained results confirm the potential of the proposed approach in terms of classification scene accuracy. It demonstrates, besides the interest of exploiting second order statistics and adopting an ensemble learning approach, the benefit of incorporating visual saliency maps

Biomonitoring heavy metals (Cu, Li and Mn) in the Marchica Lagoon of Morocco using

Concentration of Copper, Lithium and Manganese were determined in the whole soft tissues of Mytilus galloprovincialis, collected from the two sites (Bni Ansar and Kariat Arekmane) of the Marchica lagoon of Morocco. The mussels were sampled on December and July of 2019. The ability of mussels to accumulate metals was arranged in the following order: Li < Cu < Mn. The levels of heavy metals in M. galloprovincialis were higher (P<0.05) in December (7.38, 2.63 and 11.10 mg/kg d.w., for Cu, Li and Mn, respectively) than July (5.56, 1.85 and 7.24 mg/kg d.w., for Cu, Li and Mn, respectively) because of the environmental parameters of the seawater and the physiological status of the animal. The trends of accumulations of investigated metals in mussel were higher (P < 0.05) in samples from Bni Ansar than from Kariat Arekmane sites, because of the urban and industrial discharge that submitted the zone of lagoon near to the Bni Ansar city. The Mn concentration in the mussel exceeded the acceptable guidelines limits indicated by international organization, which suggests that consumption of bivalves represents a threat to human health. The studied mussel is suitable biomonitors to investigate heavy metals contamination in the coastal area of the Moroccan Mediterranean coasts